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Summary 



Description of Concern 

The ETMS program is a Naval officer postgraduate degree program which was initiated in the early 
1970's. It is offered through civilian universities, and results in a Masters of Arts degree in 
Education and the award of a Navy subspecialty code as an Education and Training specialist. 

Eight universities currently sponsor this program with various curricula, but in each case 
successful completion results in a MA in Education and subspecialty designation. The Chief of 
Naval Education and Training (CHET) office in Pensacola, Florida directs this program, creating 
education-skill requirements (ESR's) and then matching course offerings with Navy requirements 
to ensure a certain level of consistency throughout the varying degree requirements. The match 
between skill requirements and fleet needs is crucial to the success of the program. Graduates of 
the ETMS program are expected to go on in their Naval careers working in various positions in 
areas such as education and training, instructional design, and evaluation. There are currently 142 
graduates now in the fleet. Periodic evaluation of the program is required to ensure that the 
program continues to evolve in response to fleet needs. 

Recommendations 

1 . A second evaluation should be conducted targeting only and all of the XX37P coded 
officers currently in the fleet. 

2 . Naval Military Personnel Command should be alerted of the underutilization of ETMS 
officers and make adjustments in the assignment procedure to ensure that subspecialists are 
given a utilization tour within the first two tours after graduation as specified by OPNAV. 

3 . The follow up evaluation should be of sufficient size that only the responses of XX37P 
coded officers who have been or are in a utilization tour are used to determine the relevance 
of their education to fleet ETMS jobs. 

4 . Arrangements should be made with the civilian institutions to ensure that their ETMS 
program curricula meets all of the required objectives. 

5 . An optional recommendation is that an ESR course be taught at all host civilian institutions 
(or at a local base) by Navy instructors. An example of just such a course has been created 
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by an XX37P coded officer and will be taught this fall at Old Dominion University. For 
more information See Appendix E 

Rationale 

The concept of relevancy of education to job needs is of great importance to both universities and 
employers. The majority of research that supports this statement falls into two broad categories; 
graduate's evaluation of the match between curricula and required job skills, and employers 
evaluation of graduates job skills, performance, and knowledge. Studies have determined that the 
more that is known about competencies needed in careers and are taken into account in curriculum 
development the more competitive graduates could be. Educational researchers have found 
evidence that follow-up studies have great value to any college, yet they have also found that 
follow-up studies of graduates to determine their perceptions of the strengths and weaknesses of 
curricula are infrequently undertaken 

Current research demonstrates the ongoing need by both educational institutions and employers for 
continuing studies that evaluate the balance between employers needs and the curriculum provided 
for students. Such evaluations will help create a dynamic and systematic relationship between 
educational institutions and the work place. 

Implications 

There should be no major cost effects involved other than those normally accrued during the 
biannual on site curriculum evaluation. The possible exception to this being the creation of a 
"Navy ESR" course. As one ETMS officer has already created the course, it would merely 
necessitate the duplication costs and the assignment of appropriate civilians or NROTC staff to 
instruct the course at site universities. 
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PROGRAM DESCRIPTION 



Back£round^ 

The Education and Training Management Subspecialty (ETMS) program is a Navy officer postgraduate 
degree program which was initiated in the early 1970's. It is offered through various civilian universities, 
and results in the completion of a Masters of Arts degree in Education and the award of a Navy subspecialty 
code as an Education and Training specialist. Eight universities currently sponsor this program with 
various curricula, but in each case successful completion results in a MA in Education and subspecialty 
designation. The Chief of Naval Education and Training (CNET) office in Pensacola, Florida directs this 
program, matching course offerings with Navy requirements to ensure a certain level of consistency 
throughout the varying degree requirements. Graduates of the ETMS program are expected to go on in 
their Naval careers working in various positions in areas such as education and training, instructional 
design, and evaluation. There are currently 142 graduates who are out in the fleet. 

Evaluation Puyyose 

• Are the knowledge and skills defined in the Education and Training Management Subspecialty 
(ETMS) Educational Skills Requirements (ESR), acquired by program participants? 

• Are the skills and knowledge acquired with the ETMS Masters of Arts degree relevant and 
applicable to Navy education jobs in the fleet? 

• What did Navy officers like and dislike about the ETMS program, and what would they do to 
improve it? 

• And a question ultimately raised, though not addressed in this study, do Navy officers find 
certain universities’ curricula more relevant to fleet jobs than others? 

These are the questions that this evaluation hopes to provide answers and insights to. 

The primary purpose for this evaluation is to determine the level ESR knowledge acquired, and if the skills 
gained from the ETMS program are useful and applicable to actual education jobs in the fleet Additionally, 
it is hoped that information regarding the most optimal curricula/courses (or combination of 
curricula/courses) for the ETMS program would be identified. This secondary information would allow, at 
a later time, a comparison among the various universities' curricula for potential change and improvement. 
The evaluation project examined the opinions of ETMS graduates and their supervisors regarding their level 
of ESR knowledge acquisition, and the relevancy and applicability of their degree programs to their current 
jobs. Questionnaires and interviews were triangulated to access attitudes and opinions about the program. 

It is hoped that the summary of findings will be used to trigger further evaluation of individual university 
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curricula for optimal course determination, as well as be used for by decision-making groups to modify, or 
improve the existing programs to more readily meet fleet needs. 

Issues / Concerns 

The primary audience for this evaluation is CNET. Dr. J.D. Smith, head of Civilian Institutions Programs 
at CNET is the client. The secondary audience is Navy Postgraduate School at Monterey, Civilian 
Institutions coordinator, LT Padgett. Stakeholders would be the eight universities with ETMS programs: 
George Washington University, Harvard University, Memphis State University, Old Dominion University, 
San Diego State University, Stanford University, University of West Florida and recently added University 
of Rhode Island. Current and future ETMS students will also be stakeholders. 

There are political implications concerning the various civilian university curricula being scrutinized, 
however as routine changes and upgrade of the program occur based on the Navy's requirements, potential 
problems should be minimized. This evaluation should prove to be a valuable tool for the existing program 
verification system. Both clients, CNET and Navy Postgraduate School, are very supportive and interested 
in this project. 
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METHODOLOGY 



Evaluation Design 

A formative evaluation design was selected utilizing a goal oriented/decision focussed approach The goal 
oriented approach was identified because of the clearly defined objectives of the ETMS program which 
facilitate agreement scale self-report measurement This approach was not chosen singularly, however, as 
the program is ongoing and requires decision-focussed evaluation information for potential change and or 
improvement. 

A qualitative evaluation was indicated due to the extraneous variables that could influence or invalidate 
quantitative measurements. Any tangible measurement (promotion, job selection, awards etc...) of an 
ETMS officer’s competency/success in the field could not be isolated from such random variables as time in 
rank, fleet requirements, personality traits, and billet opportunities. 

Samplin g 

The respondents for the questionnaire were a sample of ETMS subspecialty Naval officers currently in the 
fleet. The initial criteria for respondent sample was a maximum of two tours of duty (up to six years) after 
graduation from the program. A stratified random sample was then taken from the remaining the 96 ETMS 
officers that met the criteria listed above. The stratification process was based on rank of officers 
(lieutenants and Lieutenant Commanders) to ensure a equally representative sample from those officers 
who met the initial criteria. Once stratified the sample was taken using the random numbers table. 40 
questionnaires were mailed out to ETMS Subspecialists. 

The respondents for the supervisors questionnaires were selected using the list of ETMS coded billets listed 
in a Navy officer billet listing publication. As only 15 billets were identified for ETMS Subspecialists only 
fifteen questionnaires were sent out, one to the supervisors at each command. 

The respondents for the interviews were randomly selected from the remaining ETMS officer population. 
Both officers were working in XX37P coded billets. Both were contacted and interviewed by phone using 
the questionnaire as a format for the closed and open response questions. 

Data Collection 

Three types of data were collected for a triangulation of information. The first type was the questionnaires 
mentioned above. These were in a semi-structured format with a predominance of agreement scale 
questions followed by open response probes at the end. The questions asked the respondents to rate their 
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acquisition level of 10 ESR’s, as identified as the Navy's ETMS program objectives, and their relevancy to 
fleet education jobs. The open-response portion of the questionnaires asked the sample for their personal 
ETMS history information, their opinions of the program, and their recommendations for program 
improvement. 

The second collection method was also agreement scale questionnaires, however these were targeted at 
fifteen supervisors that oversee ETMS officers in fleet education jobs. This questionnaire followed the 
same format as above, only supervisors rated their subordinates on their mastery of the the 10 ESR's as 
well as the supervisors rating their perceptions of skill’s relevancy to the job. It also ended with an open 
response section. 

Finally two interviews were conducted. The interview, in a highly structured format, paralleled the 
questionnaires with the emphasis on open responses to all questions. The first part of the interview used 
the same format of the agreement scale questions used above. This was followed by the same open-ended 
questions that were also used at the end of the questionnaires. 

Ijistrument Objectives 

The following questions were addressed by this evaluation study via the two questionnaires and the 
interviews: 

• Is the knowledge level specified by the ten educational skill areas acquired through the ETMS 
program? 

• Are the skills attained from the ETMS program relevant and applicable to fleet education jobs? 

• What did Navy ETMS officers like and dislike about the program and what would they do to 
improve it? 

Additional information collected at the clients request include: 

• University attended during ETMS program participation. 

• How long between graduation and utilization tour. 

• What is more career enhancing: department head or utilization tour. 

• What continuing education has been pursued. 

Data Analysis 

The data collected was analyzed by two methods. The closed response data collected in the 
questionnaires was scored and rated on knowledge level, and the applicability and relevance (both graduate 
and supervisor ratings) of the 10 skill areas to education jobs. Values were assigned to responses on a 
scale of 1 to 5. One being very low, five being very high. Number and percentage of responses were 
assigned to ESR’s by respondent category. Mean scores were also assigned for each ESR rating by 
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respondent category. The open responses were analyzed with the those from the interviews using the 
responses themselves to create categories or patterns of response. Trends were identified and then analyzed 
against the closed responses for potential commonality. 

L/m/tot/ons of Study 

The major limitation of this evaluation is it’s potential lack of generalizability. This is due to the low 
number of usable questionnaires returned by the sample. Besides the normal disinterest most people have 
for completing and returning questionnaires a secondary element was identified as the cause for low 
usability of responses. The sample was selected from a computer print out titled "XX37P Coded Officers" 
that was obtained from the Graduate Education Placement Department of the Navy Military Personnel 
Command at Washington D.C. As questionnaires were returned, it became apparent that the list sent from 
Washington contained not only XX37P coded officers, but XX37S coded officers as well. The "S" code 
identifies that the individual received their subspecialty rating not from a school program, but through 
experience alone. Table 1 demonstrates the response rates. 



Table 1 

Questionnaire Response Rate 





Sample 

Total 

N=55 


Responses 

N=27 


Percent 

Response 


ETMS Graduates 








Lieutenant Commanders 


20 


11 


55% 


Lieutenants 


20 


10 


50% 


Total 


40 


21 


55% 


(Usable) 




(15) 


(37%) 


ETMS Officer Supervisors 


15 


6 


40% 



Basically despite the fact of a 55% questionnaire return rate from the ETMS officers, after removing the 
XX37S officers from the returned data, only 37% of those returned were usable. Due to time restrictions it 
was not possible to request a second "purified" list of P coded officers and re-select the sample. 
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FINDINGS 



The following information was collected and divided into three primary sections to address the major 
evaluation questions: Is the knowledge level specified by the ten educational skill areas acquired through 
the ETMS program?; Are the skills and knowledge attained from the ETMS program relevant and applicable 
to fleet education jobs?; What did Navy ETMS officers like and dislike about the program and what would 
they do to improve it? The additional demographic information collected at the client s request can be found 
in Appendix A. 

Is the knowledge level specified by the ten ESR's acquired through the 
ETMS program? 

Educational skill requirements were rated by the sample based on level of knowledge attained and level of 
relevance to fleet jobs. The findings from the data were further divided by ETMS officers self rated 
information, and ETMS Officer Supervisor rated information. Finally all data collected, including 
interviews were compared in tables under each heading. 

Budget constraints and fleet needs dictate that the Navy educational programs not only identify specific 
knowledge required for a field, but ensure that it is acquired by program participants. Table 2 displays the 
rate of ESR knowledge acquired as identified by actual ETMS officers based on number and percent of 
responses to each knowledge acquisition level. The major portion of the respondents rated their knowledge 
acquisition of ESR’s as moderate to high. The noticeable exceptions are ESR's 2,3, and 9. All three of 
these skill requirements (Naval Education and Training organization, Planning Programming and 
Budgeting System, and Contract Administration) are very uniquely "Navy" oriented which might help 
explain the lack of this type of education in civilian university programs. This table also illustrates the 
central tendency in scoring, with a very strong inclination for a "moderate" rating average. 
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Table 2 

Level of ESR Knowledge Acquisition 
(Self-Rated) 

N=15 



Very Very No 

High High Moderate Low Low Opinion 





% 




% 




% 


% 


Educational Skill Requirement Level 


(N) 


(N) 


(N) 


(N) 


(N) 


(N) 


1. Knowledge of basic Management 


6.5% 


47% 


40% 


6.5% 


0% 


0% 


Principles: 


(1) 


(7) 


(6) 


(1) 






2. Knowledge of Naval Education and 


0% 


13% 


27% 


20% 


40% 


0% 


Training (NAVEDTRA) organization and 
operations: 




(2) 


(4) 


(3) 


(6) 




3. Knowledge of Planning, Programming 


6.5% 


0% 


20% 


40% 


27% 


6.5% 


and Budgeting System (PPBS): 


(1) 




(3) 


(6) 


(4) 


(1) 


4. Knowledge of the principles and 


27% 


13% 


20% 


20% 


20% 


0% 


applications of Instructional Systems 
Development (ISD): 


(4) 


(2) 


(3) 


(3) 


(3) 




5. Knowledge of Personnel/Manpower 


20% 


27% 


40% 


6.5% 


6.5% 


0% 


management policies and procedures: 


(3) 


(4) 


(6) 


(1) 


(1) 




6. Knowledge of training/educational 


47% 


33% 


13% 


6.5% 


0% 


0% 


psychology: 


(7) 


(5) 


(2) 


(1) 






7. Knowledge of the Training Research and 


0% 


33% 


47% 


13% 


0% 


6.5% 


Development process: 




(5) 


(7) 


(2) 




(1) 


8. Knowledge of the applications of 


13% 


20% 


40% 


13% 


13% 


0% 


computer technology to Education/Training 
management: 


(2) 


(3) 


(6) 


(2) 


(2) 




9. Knowledge on Contract Administration: 


0% 


6.5% 


20% 


40% 


27% 


6.5% 




(1) 


(3) 


(6) 


(4) 


(1) 


10. Knowledge on existing and conceptual 


13% 


6.5% 


40% 


27% 


0% 


13% 


training systems: 


(2) 


(1) 


(6) 


(4) 




(2) 



Table 3 quantifies the same type of data as rated by ETMS officer supervisors. 
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Table 3 

Level of ESR Knowledge Acquisition 
(Supervisor Rated) 

N=6 



Very Very No 

High High Moderate Low Low Opinion 



Educational Skill Requirement Level 


(N) 


% 

(N) 


(N) 


~ %— 
(N) 


— ST" 

(N) 


% 

(N) 


1 . Knowledge of basic Management 
Principles: 


67% 

(4) 


33% 

(2) 


0% 


0% 


0% 


0% 


2. Knowledge of Naval Education and 
Training (NAVEDTRA) organization and 
operations: 


50% 

(3) 


33% 

(2) 


16.5% 

(1) 


0% 


0% 


0% 


3. Knowledge of Planning, Programming 
and Budgeting System (PPBS): 


16.5% 

(1) 


50% 

(3) 


16.5% 

(1) 


0% 


16.5% 

(1) 


0% 


4. Knowledge of the principles and 
applications of Instructional Systems 
Development (ISD): 


67% 

(4) 


33% 

(2) 


0% 


0% 


0% 


0% 


5. Knowledge of Personnel/Manpower 
management policies and procedures: 


83% 

(5) 


16.5% 

(1) 


0% 


0% 


0% 


0% 


6. Knowledge of training/educational 
psychology: 


83% 

(5) 


16.5% 

(1) 


0% 


0% 


0% 


0% 


7. Knowledge of the Training Research and 
Development process: 


67% 

(4) 


16.5% 

(1) 


16.5% 

(1) 


0% 


0% 


0% 


8. Knowledge of the applications of 
computer technology to Education/Training 
management: 


16.5% 

(1) 


50% 

(3) 


16.5% 

(1) 


16.5% 

(1) 


0% 


0% 


9. Knowledge on Contract Administration: 


16.5% 

(1) 


16.5% 

(1) 


16.5% 

(1) 


16.5% 

(1) 


33% 

(2) 


0% 


10. Knowledge on existing and conceptual 
training systems: 


67% 

(4) 


16.5% 

(1) 


16.5% 

(1) 


0% 


0% 


0% 



This table shows that the majority of supervisors rate their ETMS officers as having a moderate to very 
high level of ESR knowledge acquisition. Only one skill area was rated below this, skill requirement 
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number 9, which was rated low to moderate. As stated before, Navy Contract Administration can be very 
specific in terms of Navy regulations which would make certain aspects of it difficult to be offered as a 
regular course at a civilian university. It is also apparent that supervisors rate the knowledge of their ETMS 
officers more highly than the actual officers themselves. Table 4 is an abbreviated summary of the number 
and percentages of the responses for all of the respondents as a comparison of rating levels. 

Table 4 

Knowledge Acquisition: Abbreviated Response Summary 
ETMS officers, Interviewees, and Supervisors 



ESR # and Respondents 


High to Very 
High 
% (N) 


Moderate 

% (N) 


Low to Very Low 

% (N) 


No Opinion 

%(N) 


ESR # 1. ETMS officers 


53.5% (8) 


40% (6) 


6.5% (1) 


0% 


Interviewees 


100% (2) 


0% 


0% 


0% 


Supervisors 


100% (6) 


0% 


0% 


0% 


ESR ft 2. ETMS officers 


13% (2) 


27% (4) 


60% (9) 


0% 


Interviewees 


0% 


50% (1) 


50% (1) 


0% 


Supervisors 


83% (5) 


16.5% (1) 


0% 


0% 


ESR ft 3. ETMS officers 


6.5% (1) 


20% (3) 


67% (10) 


6.5% (1) 


Interviewees 


100% (2) 


0% 


0% 


0% 


Supervisors 


67% (4) 


16.5% (1) 


16.5% (1) 


0% 


ESR ft 4. ETMS officers 


40% (6) 


20% (3) 


40% (6) 


0% 


Interviewees 


100% (2) 


0% 


13% (2) 


0% 


Supervisors 


100% (6) 


0% 


0% 


0% 


ESR ft 5. ETMS officers 


47% (7) 


40% (6) 


13% (2) 


0% 


Interviewees 


100% (2) 


0% 


0% 


0% 


Supervisors 


100% (6) 


0% 


0% 


0% 


ESR #6. ETMS officers 


80% (12) 


13% (2) 


6.5% (1) 


0% 


Interviewees 


100% (2) 


0% 


0% 


0% 


Supervisors 


100% (6) 


0% 


0% 


0% 


ESR #7. ETMS officers 


33% (5) 


47% (7) 


13% (2) 


6.5% (1) 


Interviewees 


50% (1) 


50% (1) 


0% 


0% 


Supervisors 


83% (5) 


165% (1) 


0% 


0% 


ESR ft 8. ETMS officers 


33% (5) 


40% (6) 


27% (4) 


0% 


Interviewees 


0% 


100% (2) 


0% 


0% 


Supervisors 


67% (4) 


16.5% (1) 


16.5% (1) 


0% 


ESR ft 9. ETMS officers 


65% (1) 


20% (3) 


67% (10) 


6.5% (1) 


Interviewees 


50% (1) 


50% (1) 


0% 


0% 


Supervisors 


33% (2) 


16.5% (1) 


67% (3) 


0% 


ESR ft 10. ETMS officers 


20% (3) 


40% (6) 


27% (4) 


13% (2) 


Interviewees 


50% (1) 


50% (1) 


0% 


0% 


Supervisors 


83% (5) 


16.5% (1) 


0% 


0% 
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Here by including the interviews, the wide range of response is evident. A better way to view this 
information is using the mean scores assigned for all three sample groups, as shown in Figure 1. An 
interesting feature identified here is that the interview responses are much closer in rating value to the 
supervisors than that of the rest of the ETMS officers who responded to the questionnaires. 

Figure 1 



Level of Knowledge of Educational Skill Requirements 



Very High 



High 



Moderate 



Low 



Very Low 




JB Supervisors Questionnaires 



Educational Skill Requirements 



P51 ETMS Questionnaires 
ETMS Interviews 



This figure demonstrates the vast variation of opinions on skill requirement acquisition. Because of the 
limited number of responses on which it is based, it is very difficult to identify any commonality in the 
ratings. Those skill areas with the most obvious similarity in means are ESR 6 - Education and Training 
psychology, rated high to very high, and ESR 7 - Training Research and Development Processes rated 
moderate to high. The overall mean for the ESR knowledge level is 3.6 or moderate to high. 

Are the skills and knowledge attained from the ETMS program relevant 
and applicable to fleet education jobs? 

The second major area of information collected was on the perceived level of ESR relevancy to fleet jobs. 
The first part of this discussion of findings will focus in on ESR relevancy as rated on the ETMS officers 
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questionnaires. This area of report had a great potential for variability due to the varying jobs and degrees 
of skills required in each. 

Table 5 

Level of ESR Relevancy to Fleet Job 
(Self Rated) 

N=15 



Very Very No 

High High Moderate Low Low Opinion 



Educational Skill Requirements 


% 

(N) 


% 

(N) 


fo 

(N) 


% 

(N) 


~ 3T“ 

(N) 


* 1 

(N) 


1. Relevance of basic Management 


27% 


47% 


20% 


6.5% 


0% 


0% 


Principles to the job: 


(4) 


(7) 


(3) 


(1) 






2. Relevance of Naval Education and 


20% 


13% 


27% 


13% 


33% 


0% 


Training (NAVEDTRA) organization and 


(3) 


(2) 


(4) 


(2) 


(5) 




operations to the job: 














3. Relevance of Planning, Programming 


0% 


27% 


27% 


20% 


20% 


6.5% 


and Budgeting System (PPBS) to the job: 




(4) 


(4) 


(3) 


(3) 


(1) 


4. Relevance of the principles and 


27% 


13% 


6.5% 


47% 


6.5% 


0% 


applications of Instructional Systems 


(4) 


(2) 


(1) 


(7) 


(1) 




Development (ISD) to the job: 














5. Relevance of Personnel/Manpower 


40% 


40% 


13% 


6.5% 


0% 


0% 


management policies and procedures to the 


(6) 


(6) 


(2) 


(1) 






job: 














6. Relevance of training/educational 


20% 


47% 


20% 


0% 


13% 


0% 


psychology to the job: 


(3) 


(7) 


(3) 




(2) 




7. Relevance of the Training Research and 


0% 


27% 


27% 


27% 


13% 


6.5% 


Development process to the job: 




(4) 


(4) 


(4) 


(2) 


(1) 


8. Relevance of the applications of 


13% 


47% 


13% 


13% 


13% 


0% 


computer technology to Education/Training 


(2) 


(7) 


(2) 


(2) 


(2) 




management to the job: 














9. Relevance on Contract Administration to 


6.5% 


27% 


6.5% 


20% 


33% 


6.5% 


the job: 


(1) 


(4) 


(1) 


(3) 


(5) 


(1) 


10. Relevance on existing and conceptual 


13% 


6.5% 


40% 


13% 


13% 


13% 


training systems to the job: 


(2) 


(1) 


(6) 


(2) 


(2) 


(2) 



This table demonstrates that for nine out of ten of the skill areas, more than half of the respondents rated 
them very highly relevant. The exception being Contract Administration which was rated by 53% as low to 
very low in job relevance. This table especially interesting with it's consistent polarity of the ETMS officer 
responses. This could be due to the fact that just less than half (as identified later on in this report) of the 
respondents had ever even worked in an XX37P (ETMS) coded billet The overall mean average for all ten 
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skill areas was 3.12, or slightly higher than a moderate level of job relevancy. Table 6 lists the responses 

on relevancy as perceived by ETMS Officer supervisors. 

Table 6 

Level of ESR Relevancy to Fleet Job 
(Supervisor Rated) 

N=6 



Very Very No 

High High Moderate Low Low Opinion 



Educational Skill Requirements 


% 

(N) 


% 

(N) 


% 

(N) 


% 

(N) 


% 

(N) 


% 

(N) 


1 . Relevance of basic Management 
Principles to the job: 


83% 

(5) 


16.5% 

(1) 


0% 


0% 


0% 


0% 


2. Relevance of Naval Education and 
Training (NAVEDTRA) organization and 
operations to the job: 


50% 

(3) 


16.5% 

(1) 


16.5% 

(1) 


0% 


16.5% 

(1) 


0% 


3. Relevance of Planning, Programming 
and Budgeting System (PPBS) to the job: 


16.5% 

(1) 


33% 

(2) 


16.5% 

(1) 


16.5% 

(1) 


16.5% 

(1) 


0% 


4. Relevance of the principles and 
applications of Instructional Systems 
Development (ISD) to the job: 


16.5% 

(1) 


33% 

(2) 


0% 


33% 

(2) 


16.5% 

(1) 


0% 


5. Relevance of Personnel/Manpower 
management policies and procedures to the 
job: 


50% 

(3) 


50% 

(3) 


0% 


0% 


0% 


0% 


6. Relevance of training/educational 
psychology to the job: 


67% 

(4) 


33% 

(2) 


0% 


0% 


0% 


0% 


7. Relevance of the Training Research and 
Development process to the job: 


0% 


0% 


50% 

(3) 


33% 

(2) 


16.5% 

(1) 


0% 


8. Relevance of the applications of 
computer technology to Education/Training 
management to the job: 


0% 


16.5% 

(1) 


33% 

(2) 


16.5% 

(1) 


33% 

(2) 


0% 


9. Relevance on Contract Administration to 
the job: 


33% 

(2) 


16.5% 

(1) 


0% 


50% 

(3) 


0% 


0% 


10. Relevance on existing and conceptual 
training systems to the job: 


16.5% 

(1) 


33% 

(2) 


16.5% 

(1) 


16.5% 

(1) 


16.5% 

(1) 


0% 



This table once again shows that Supervisors (with two exceptions) consistently rate ESR’s higher than 
their ETMS officers. The majority of the skill areas were rated at a moderate to very high level of relevance 
to fleet jobs. The overall mean score for all ten of the ESR's was 3.5 or moderate to high. The exception 
to this were ESR 7 - Training Research and Development, and ESR 8 - Computer Technology. Both were 
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rated as low in job relevance. Table 7 once again summarizes all the responses from the sample on on ESR 
relevancy to fleet jobs. 



Table 7 

ESR Relevancy to Job: Abbreviated Response Summary 
ETMS officers, Interviewees, and Supervisors 



ESR tt and Respondents 


High to Very 
High 

% (N) 


Moderate 
% (N) 


Low to Very Low 
% (N) 


No Opinion 

% (N) 


ESR tf 1 . ETMS officers 


73% (11) 


20% (3) 


65% (1) 


0% 


Interviewees 


100% (2) 


0% 


0% 


0% 


Supervisors 


100% (6) 


0% 


0% 


0% 


ESR ft 2. ETMS officers 


33% (5) 


27% (4) 


47% (7) 


0% 


Interviewees 


50% (1) 


0% 


50% (1) 


0% 


Supervisors 


67% (4) 


16.5% (1) 


16.5% (1) 


0% 


ESR tt 3. ETMS officers 


27% (4) 


27% (4) 


40% (6) 


6.5% (1) 


Interviewees 


100% (2) 


0% 


0% 


0% 


Supervisors 


50% (3) 


0% 


50% (3) 


0% 


ESR tt 4 . ETMS officers 


40% (6) 


6.5% (1) 


53% (8) 


0% 


Interviewees 


50% (1) 


0% 


50% (1) 


0% 


Supervisors 


50% (3) 


0% 


50% (3) 


0% 


ESR tt 5. ETMS officers 


80% (12) 


13% (2) 


6.5% (1) 


0% 


Interviewees 


100% (2) 


0% 


0% 


0% 


Supervisors 


100% (6) 


0% 


0% 


0% 


ESR #6. ETMS officers 


67% (10) 


20% (3) 


13% (2) 


0% 


Interviewees 


100% (2) 


0% 


0% 


0% 


Supervisors 


100% (6) 


0% 


0% 


0% 


ESR ttl . ETMS officers 


27% (4) 


27% (4) 


40% (6) 


6.5% (1) 


Interviewees 


50% (1) 


0% 


50% (1) 


0% 


Supervisors 


0% 


50% (3) 


50% (3) 


0% 


ESR tt 8. ETMS officers 


60% (9) 


13% (2) 


27% (4) 


0% 


Interviewees 


50% (1) 


0% 


50% (1) 


0% 


Supervisors 


16.5% (1) 


33% (2) 


50% (3) 


0% 


ESR tt 9. ETMS officers 


33% (5) 


6.5% (1) 


53% (8) 


65% (1) 


Interviewees 


100% (2) 


0% 


0% 


0% 


Supervisors 


50% (3) 


0% 


50% (3) 


0% 


ESR tt 10. ETMS officers 


20% (3) 


40% (6) 


27% (4) 


13% (2) 


Interviewees 


50% (1) 


0% 


50% (1) 


0% 


Supervisors 


50% (3) 


16.5% (1) 


33% (2) 


0% 



This table points out that most of the respondents (again with two exceptions) rated ESR to job relevancy as 
moderate to very high. It is also more consistent and shows more commonality in opinion than the earlier 
representation on knowledge acquisition. This is even more apparent in Figure 2, which is a bar graph of 
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the ESR means for each set of respondents. This chart graphically demonstrates the particular similarity in 
ratings of most of the skill areas except ESR's 3 and 9. 



Figure 2 

Level of Relevance of Educational Skill Requirements 



Very High 



High 



Moderate 



Low 



Very Low 




Q S u pe rvi so rs Q uesti o n nai res 



Educational Skill Requirements 



fvT] ET MS Questionnaires 




ETMS Interviews 



The commonality shared in the majority of ratings of the ESR’s job relevance seem to point to it as being 
more universally measured by both ETMS officers and their Supervisors. The overall mean for the 10 skill 
areas is a 3.5, or a moderate to high relevancy factor. The two low to moderate rated areas were Training 
Research and Development (ESR 7), and Computer Technology Applications (ESR 8). 



Program Positive/Negative Comments and Recommended 
Improvements 

The final section of the findings and the questionnaire/interviews related to the perceived "good and bad” 
aspects of the ETMS program. The following tables summarize the most frequently cited positive and 
negative open-responses at the end of the questionnaires and interviews. For a complete summary of all 
responses, see Appendix C. 
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Table 8 

Positive ETMS Program Open Responses 



1 . 


(N) 

2 


Interesting subject variety and selection 


2. 


2 


Good adult learning and educational psychology courses 


3. 


2 


Outstanding ISD instruction 


4. 


2 


Business courses were the most valuable 


5. 


2 


Graduates have an extensive background in Educational Training and Management 



The second and third responses are useful in that they support data collected in the agreement scale portion 
of the questionnaires and interviews. Response number two directly reflects the overall mean rating of 
knowledge acquisition of high to very high in ESR 6- Training and Educational psychology. Response 
number three is interesting in that it coincides with the overall mean rating of high in knowledge acquisition 
for ESR 4. Response number five lightly contradicts the overall mean rating of moderate to high for the 
knowledge acquisition of ESR 1, Management Principles. 



Table 1 1 lists the most frequently cited negative responses about the ETMS program. 



Table 9 

Negative ETMS Program Open Responses 



(N) 



1. 5 

2. 4 

3. 4 

4. 3 

5. 3 

6. 3 



Many course options aren't relevant to program (ie. School Law, Computer Programming, 
Client/consultant courses) 

University ETMS program advisor is not helpful 

Courses geared strictly to civilian programs/needs (primarily teachers and principals) 

No courses on Contract Administration - need for job 
Management courses redundant for Naval Officers 

Frequently courses most relevant to ESR's no offered (during program completion time) 



The comments made in response one, that courses are frequently not applicable to the program (computer 
programming is one of the courses cited), supports the overall low to moderate relevancy rating of ESR 8 - 
Computer Technology. On the other hand it does not support the low to moderate rating of knowledge 
level for the same skill area. Response four about Contract Administration supports the overall very low to 
low rating for knowledge level of ESR 9, and becomes an important statement when compared to an overall 
rating of moderate to high in job relevancy. Response number five on management courses, is balanced by 
an overall rating for ESR 1 (Management Principles) of Moderate to High in knowledge level but a high to 
very high level of relevancy. The final response is especially interesting in that it supports the low to 
moderate rated knowledge level ESR’s (2- NAVEDTRA Organization, 3- Planning Programming and 



15 



Budgeting System and 9- Contract Administration). The last response validates the need for courses that 
meed the curricula standards for ESR's 2,3 , and 9. 

This last table summarizes the most frequently mentioned comments on changes to improve the program 
that the respondents said that they would like to see. As above a complete summary of all responses can be 
found in Appendix C. 



Table 10 

Recommended Change Responses to Improve ETMS Program 





(N) 


Add courses on: 


1 . 


4 


Contract Administration 


2. 


3 


Program evaluation 


3. 


2 


How Navy training courses curriculum are designed and developed 


4. 


2 


ISD (specially test development and objective writing 


5. 


2 


Navy Training ESR course 


6. 


2 


Educational Technology (development and training aids) 



The most frequently cited responses all involve course additions. While responses one and three are 
supported by previously mentioned low to moderate rated knowledge level ESR's, responses four and six 
are contradicted by the low to moderate job relevancy of their related ESR's. 
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CONCLUSIONS & RECOMMENDA TIONS 



1. Based on the small size of the useable 
response from the sample, this evaluation 
may not be generalizable to the entire 
ETMS program and it's community of 
subspecialists. 

2. More than half of the responses were 
received from ETMS officers who have 
never been in a utilization tour. 



3. Three Educational Skill Requirements 
(Contract Administration, Personnel 
Planning and Budgeting System, and 
NAVEDTRA Organization) are not 
sufficiently addressed by most civilian 
universities. 



A secondary evaluation should be conducted 
targeting only and all of the XX37P coded 
officers. 



Naval Military Personnel Command should be 
alerted to this and make adjustments in the 
assignment procedure to ensure that 
subspecialists are given a utilization tour within 
the first two tours after graduation as specified 
by OPNAV. 

The follow up evaluation should be of 
sufficient size that only the responses of 
XX37P coded officers who have been or are in 
a utilization tour are used to determine the 
relevance of their education to fleet ETMS jobs. 

Arrangements should be made with the civilian 
institutions to ensure that their ETMS program 
curricula meets all of the required objectives. 

An optional recommendation is that an ESR 
course be taught at all host civilian institutions 
(or at a local base) by Navy instructors. An 
example of just such a course has been created 
by and XX37P coded officer and will be taught 
this fall at Old Dominion University. For more 
information See Appendix E. 
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LITERATURE REVIEW 



The concept of relevancy of education to job needs is of great importance to both universities and 
employers. The majority of literature that supports this statement falls into two broad categories; graduate's 
evaluation of the match between curricula and required job skills, and employers evaluation of graduates 
job skills, performance, and knowledge. 

Wheelock and Zekeri (1988) conducted a study that attempted to determine which competencies former 
students found most essential to their careers. They hypothesized that the more that is known about 
competencies needed in careers and are taken into account in curriculum development the more competitive 
graduates could be. According to Cedergrin ( 1985) continuing follow-up has great value to any college, 
yet he feels that follow-up studies of graduates to determine their perceptions of the strengths and 
weaknesses of curricula are infrequently undertaken. In his book, Konrad (1975) expressed the opinion 
that data that are generated from follow-up studies should be utilized as soon as they are available and then 
stored in a management information system until further analysis can be undertaken. In an article by Hoyt 
(1987), results from a 1987 Gallup poll are cited that indicate only about half of the employed college 
graduates said that their skills and abilities were being used "very well" in their current jobs. Houghton's 
studies (1971) of the business program at Southern Oregon College found that the weakest aspects of the 
program in the opinion of the graduates were that the courses were too theoretical and the curriculum was 
poor. In another recent study (Gerbert 1988) surveys were used to identify the topics that new dentists 
believed to be under-emphasized in the dental school curriculum as well as those that are most important to 
dental practice. 

Camoy (1979) researched the role of education in meeting employment objectives in the hopes of 
identifying policy oriented planning procedures to ensure employer needs are met Also hoping that the 
results of study would help improve educational programs to better meet employers' needs, Travis and 
Jones ( 1976) had supervisors evaluate graduates in the area of job performance. In a another study to 
determine how adequately college prepared 1978 graduates for employment in some specific job skill and 
areas of knowledge, Campbell and Dogoloff (1979) found that employers rated 94% of graduates 
performance of job skills as "adequate or more than adequate." Additionally, the survey results indicated 
that while supervisors rated the overall job performance of graduates as adequate or more than adequate, 
they cited the need for improvement in technical and problem solving skills. 

These studies demonstrate the ongoing need by both educational institutions and employers for continuing 
research that evaluates the balance between employers needs and the curriculum provided for students. 

Such evaluations will help create a dynamic and systematic relationship between educational institutions and 
the work place. 
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Appendix A. - Cljent Re£uested D emograp/i/cs 



A - 1 



This next section of findings will report the information collected specifically at the clients request. 

Current Billet 

The first question to be addressed is the utilization of ETMS officers in specific XX37 coded billets. 

Figure 3 is a pie chart representation of the respondents (ETMS officers only) cited tour utilization history. 
It reveals several interesting trends. The foremost being that nearly half (41 %) of the respondents have not 
had a utilization tour to date. This is especially alarming in that it identifies the underutilization of a readily 
available and costly asset: Navy officers with masters degrees in education that were frequently fully 
funded by the government. 



ETMS Officer XX37P Billet Utilization 




m Current Tour 41% (7) 
Previous Tour 20% (3) 
1 Never 41% (7) 



Of the 17 ETMS Officer responses (including the interviewees) six were placed in utilization tours 
immediately after graduation, four on their second tour after graduation. As noted above the remaining 
seven have never had a utilization tour. Personal preference may be a factor in this, as officers frequently 
choose billets outside of their subspecialty for various reasons. 

Universities Attended 

The following table identifies the respondents by the universities they attended during their participation in 
the the ETMS program. Only five of the possible eight universities are represented in the sample, but each 
is relatively equal in representation. 



A -2 




Table 11 

Breakdown of ETMS Graduate Respondents 
by University 



N=17* 



Stanford 


George 

Washington 


West Florida 


Old Dominion 


San Diego 
State 


%(N) 


%(N) 


%(N) 


%(N) 


%(N) 


20% (3) 


27% (4) 


27% (4) 


27% (4) 


13% (2) 



Continuing Education 

Of the 17 ETMS officers (including interviewees) only 2 had, or are pursuing continuing education. Of 
those two, one is taking courses in Education and Human Resources Development, the other has completed 
a doctorate in Education Administration. 



Career Enhancement 

This question is very pertinent as the Navy continues to shift back and forth from a strictly leadership track 
to one more based on a subspeciality career emphasis. It is interesting to note that the majority of both the 
ETMS officers and their supervisors cited the Department Head tour as more career enhancing. The 
comments made by the three respondents that answered out of category, were particularly helpful in 
identifying what is perceived as "the best of both worlds", the optimum answer a dual Department 
Head/Utilization tour. Unfortunately this type of billet is considerably rare in ETMS assignments. 

Table 12 

ETMS Officer and Supervisor 
Career Concepts 
N=23* 



Most Career Enhancing 
after Graduation? 


Subspecialty Utilization 
Tour 

% (N) 


Department Head 
Tour 
% (N) 


ETMS officers 


27% (4) 


60% (9) 


ETMS Officer Interviewees 


0% 


100% (2) 


ETMS Officer Supervisors 


17% (1) 


66% (4) 


Total 


35% (5) 


52% (15) 
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* Two of the ETMS officers and one Supervisor wrote in that the "ideal career enhancing tour" after 
graduation would be a Department Head/Utilization tour. 
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Appendix B. - Questionnaires 



ETMS Officer Questionnaire 



Try to remember back, and based solely on your ETMS degree education for the XX37P subspecialty, 
please rate the the following Education Skill Requirements by the level of knowledge you acquired and it’s 
applicability and relevance to your current job. Please read each skill area and circle the level that most 
closely matches yours. There are six possible responses: Very High, High, Moderate, Low, Very Low, 
and No Opinion. 

At the end there are eight general information questions, followed by space for your recommendations on 
improving the ETMS program. Feel free to add any additional comments that you feel would be pertinent. 
Thank you again for your time. 



1 . Level of knowledge of basic Management Principles: 

• fundamental management theories 

• management system basic elements 

• resource management system 

2. Level of relevance of basic Management Principles to 
your job. 

3. Level of knowledge of Naval Education and Training 
(NAVEDTRA) organization and operations: 

• organizational structure 

• training program implementation procedures 

• training program maintenance 

• training requirement validation 

• education and training roles of Navy commands 

4. Level of relevance of NAVEDTRA organization and 
operations knowledge to your job. 

5. Level of knowledge of Planning, Programming and 
Budgeting System (PPBS): 

• translating validation into acquisition 

6. Level of relevance of PPBS to your job. 

7. Level of knowledge of the principles and applications of 
Instructional Systems Development (I SD): 

• Tenets of instructional technology and ISD 

• ISD advantages 

• instructor roles with ISD 

8. Level of relevance of ISD to your job. 



No Opinion 
No Opinion 
No Opinion 
No Opinion 
No Opinion 

Very Low No Opinion 



Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 



Very High High Moderate Low Very Low 

Very High High Moderate Low Very Low 

Very High High Moderate Low Very Low 

Very High High Moderate Low Very Low 

Very High High Moderate Low Very Low 



Very High High Moderate Low 
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9. Level of knowledge of Personnel/Manpower management 
policies and procedures: 

• theories of manpower management 

• tenets of personnel administration 

10. Level of relevance of basic Personnel/Manpower 
management policies and procedures to your job. 

1 1. Level of knowledge of training/educational psychology: 

• learning processes 

• learning theory 

12. Level of relevance of Training/Educational Psychology 
to your job. 

13. Level of knowledge of the Training Research and 
Development process: 

• steps for the research and development process 

• research and development initiation and guidance 

14. Level of relevance of the Training Research and 
Development process to your job. 

15. Level of knowledge of the applications of computer 
technology to Education/Training management: 

• ADP systems 

• computer theory and system analysis 

• management of computer applications 

• computer capability /limitations 

• computer assisted instruction 

16. Level of relevance of computer technology knowledge to 
your job. 

17. Level of knowledge on Contract Administration: 

• procurement management 

• procurement regulations 

• product evaluation 

18. Level of relevance of Contract Administration to your 
job. 



Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 
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19. Level of knowledge on existing and conceptual training 
systems: 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


• non ISD training theories: assets/limitations 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


20. Level of relevance of knowledge on existing and 
conceptual training systems to your job. 

21. Are you currently in an XX37P coded billet? 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 



• If no, when were you last in an XX37P coded billet? 

22. What university did you attend during your participation in the ETMS program? 

23. How long was it after graduation before your XX37P utilization tour? 

24. Have you pursued continuing education since graduation? 

• If yes, what courses and in what field? 



25. What would have been more career enhancing for you after graduation; a department head or utilization tour? 

26. Was there anything that you especially liked about the curricula offered at your university for ETMS program? 



27. Was there anything that you especially disliked about the curricula offered at your university for ETMS program? 
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28. If you could, what changes/improvements would you make to the ETMS program, or the university's curriculum to make 
it more job relevant? 



Please feel free to make any additional comments. 



Thank you again for your help. 
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ETMS Supervisor Questionnaire 



Based on your knowledge of and experience with the XX37P officer who works for you, please rate 
him/her on the following Education Skill Requirements by the level of knowledge displayed to you. Also 
please rate the level of applicability and relevance of these skills to his/her job. Please read each skill area 
and circle the level that most closely matches. There are six possible responses: Very High, High, 
Moderate, Low, Very Low, and No Opinion. 

At the end there are six general information questions, followed by space for any recommendations you 
may have for improving the ETMS program/curricula Feel free to add any additional comments that you 
feel would be pertinent Thank you again for your time. 



1. Level of knowledge of basic Management Principles: 

• fundamental management theories 

• management system basic elements 

• resource management system 

2. Level of relevance of basic Management Principles to the 
job. 

3. Level of knowledge of Naval Education and Training 
(NAVEDTRA) organization and operations: 

• organizational structure 

• training program implementation procedures 

• training program maintenance 

• training requirement validation 

• education and training roles of Navy commands 

4. Level of relevance of NAVEDTRA organization and 
operations knowledge to the job. 

5. Level of knowledge of Planning, Programming and 
Budgeting System (PPBS): 

• translating validation into acquisition 

6. Level of relevance of PPBS to the job. 

7. Level of knowledge of the principles and applications of 
Instructional Systems Development (ISD): 

• Tenets of instructional technology and ISD 

• ISD advantages 

• instructor roles with ISD 

8. Level of relevance of ISD to the job. 



Opinion 

Opinion 

Opinion 

Opinion 

Opinion 

Opinion 



Very High 


High 


Moderate 


Low 


Very Low 


No 


Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No 


Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No 


Opinion 


Very High 


High 
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Low 


Very Low 


No 
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No 
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Low 
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No 
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Very Low 


No 


Very High 


High 
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Very Low 


No 


Very High 


High 


Moderate 


Low 


Very Low 


No 


Very High 


High 


Moderate 


Low 


Very Low 


No 


Very High 


High 


Moderate 


Low 


Very Low 


No 
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9. Level of knowledge of Personnel/Manpower management 
policies and procedures: 

• theories of manpower management 

• tenets of personnel administration 

10. Level of relevance of basic Personnel/Manpower 
management policies and procedures to the job. 

11. Level of knowledge of training/educational psychology : 

• learning processes 

• learning theory 

12. Level of relevance of Training/Educational Psychology 
to the job. 

13. Level of knowledge of the Training Research and 
Development process: 

• steps for the research and development process 

• research and development initiation and guidance 

14. Level of relevance of the Training Research and 
Development process to the job. 

15. Level of knowledge of the applications of computer 
technology to Education/Training management: 

• ADP systems 

• computer theory and system analysis 

• management of computer applications 

• computer capability/limitations 

• computer assisted instruction 

16. Level of relevance of computer technology knowledge to 
the job. 

17. Level of knowledge on Contract Administration: 

• procurement management 

• procurement regulations 

• product evaluation 

18. Level of relevance of Contract Administration to the job. 



Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 


Very High 


High 


Moderate 


Low 


Very Low 


No Opinion 
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19. Level of knowledge on existing and conceptual training Very High High Moderate Low Very Low No Opinion 

systems: 

• non ISD training theories: assets/limitations Very High High Moderate Low Very Low No Opinion 

20. Level of relevance of knowledge on existing and Very High High Moderate Low Very Low No Opinion 

conceptual training systems to the job. 

21. Do you know what university your XX37P officer attended during his/her participation in the ETMS program? 

22. Do you know if your XX37P officer is/has pursued continuing education since graduation? 

• If yes, do you know what courses and in what field? 

23. What do you feel is more career enhancing for an XX37P officer for after graduation; a department head or utilization tour? 

24. Is there anything that positively stands out about your XX37P officer's knowledge and ability? 



25. Is there anything that negatively stands out about your XX37P officer's knowledge and ability?? 
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26 . If you could, what changes/improvements would you make to the ETMS program, or curriculum to make it more job 
relevant? 



Please feel free to make any additional comments. 



Thank you attain for your help. 
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Appendix C. - Questionnaire and /nterv/ew Open Re sponse Summary 
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N=23 



tt Positive ETMS Program Responses 

II Interesting subject variety and selection 
II Good adult learning and educational psychology courses 
II Outstanding ISD instruction 

II Business courses were the most valuable 

II Graduates have an extensive background in Educational Training and Management 
PPBS course offered was relevant and useful 
Practicum for Principals was the best course 
It made me computer literate (finally) 

Outstanding instructors 

ODU program is more adult leaming/HRD thus more relevant than other programs 
Economics of Education course was great 

Teacher Evaluation course was very relevant for Navy instructor evaluations 
Good educational administration, personnel and organizational change courses 
Statistics course was very useful to my current job. 

Excellent cognitive learning theory 



§ Negative ETMS Program Responses 

IV University ETMS program advisor is not helpful 

IV Courses geared strictly to civilian programs/needs (primarily teachers and principals) 

V Many course options in the program are not applicable (ie. School Law, Computer 
Programming, Client/consultant courses 

III No courses on Contract Administration - need for job 

ID Management courses redundant for Naval Officers 

H Frequently courses most relevant to ESR's no offered (during program completion time) 
Most of what we learned is not relevant or used on the job 
XX37P officers not fully utilized in fleet jobs 
Insufficient time to complete the program 

Civilian instructors have no experience with Navy Education an Training 
ETMS curriculum is disorganized 
No mentors available 
Business courses very hard to enroll in 
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ft Recommended Change Responses to Improve ETMS Program 

Add courses on: 

Contract Administration 
Program evaluation 

How Navy training courses curriculum are designed and developed 
ISD (specially test development and objective writing 
Navy Training ESR course 

Educational Technology (development and training aids) 

3 or 4 elective courses for area of concentration (ED or HRD) 
Training research and development 
Front end analysis 
II Allow more time for program completion 



ft General Respondent Recommendations for XX37's 

Create a specific XX37 career progression 

Ensure first tour is a utilization tour 

Ensure First tour is a "schools house" tour 

CNET should track XX37's to ensure maximum utilization tours 
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Appendix D. - Cover Letters 
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9 April 1991 



Dear Sir, 

I am a graduate student in the Education and Training Management Subspecialty (ETMS) program 
at San Diego State University. Currently I am working on my masters project, an evaluation of the ETMS 
program, for Dr. J.D. Smith at the office of the Chief of Naval Education and Training (CNET). The focus 
of this evaluation is to determine the level to which various civilian universities' curricula help ETMS 
subspecialists (XX37P coded officers) acquire knowledge from the ten education skill areas defined by 
CNET, and the relevance and applicability of this knowledge to an XX37P officers job in the fleet. Your 
input is vital to this evaluation. 

Enclosed is a short questionnaire, and a return addressed envelope. It should take approximately 10 
to 15 minutes to complete. If you do not directly oversee the XX37P officer at you command, please ask 
the responsible department head to complete and return it as soon as possible. If there are more than one 
XX37P officer at you command, if you wish, please make copies of this questionnaire to accommodate 
them. If you would like a copy of the finished evaluation summary, just write a note at the end of the 
questionnaire. 

This evaluation will not be valid without your input. Thank you in advance for you assistance. 

With your participation, this evaluation will help to improve the ETMS program for the fleet and future 
graduates. 



Very respectfully, 
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9 April 1991 



Dear ETMS Subspecialist, 

I am an ETMS graduate student currently working on my masters project at San Diego State 
University. The project is an evaluation of the ETMS program, for Dr. J.D. Smith at CNET. The focus of 
the evaluation is to determine the level to which the various universities' curricula helped you acquire the 
knowledge from the ten education skill areas, and the relevance and applicability of this knowledge to your 
job as an XX37P. Your input is vital to this evaluation. 

Enclosed is a short questionnaire, and a return addressed envelope. It should take you 
approximately 10 to 15 minutes to complete it If you are not currently in a XX37P coded billet, please 
answer the questions relating them to your last utilization tour. Please return it as soon as possible. If you 
would like a copy of the finished evaluation summary, just write a note at the end of the questionnaire. 

This evaluation will not be valid without your input. Thank you in advance for you assistance and 
expediency. With your participation, this evaluation will help to improve the ETMS program for future 
graduates. 



Very respectfully, 




LT USN 
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Appendix E. - ESR Course Addition inlormation 



FALL SEMESTER 1991 
ETMS STUDENTS 



ELS 697 Topics: Perspectives of Navy Training 

Index Number: 58308 

Wednesday, 4:20 - 6:50, Batten Arts & Letters Building, Room 331 



This course is designed to incorporate many of the education and 
training objectives in the Education, Training, and Management 
Subspecialty (ETMS) curriculum offered at ODU into a concise 13 
week, three hour course. A seminar approach will be used with much 
of the material presented by guest lecturers, and tours of local 
training facilities and commands. 

General Objectives : 

1. Develop an understanding of the various innovative appli- 
tions to Navy training and appropriate uses of these 
approaches. 

2. Understand the Navy reguirements for training in the 21st 
century and new approaches to meet those needs. 

3. Become familiar with the Navy's ETMS billet availabilities 
and selected career assignments. 

4. Consider potential applications of the ETMS training 
received through the university assignments. 

5. Be aware of selected local facilities and each command's 
approach towards fulfilling its mission and supporting Navy 
training. 

All ETMS students should plan take this course as part of their 36 
hour program. 
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